Dynamic language model

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for speech recognition. One of the methods includes receiving a base language model for speech recognition including a first word sequence having a base probability value; receiving a voice search query associated with a query context; determining that a customized language model is to be used when the query context satisfies one or more criteria associated with the customized language model; obtaining the customized language model, the customized language model including the first word sequence having an adjusted probability value being the base probability value adjusted according to the query context; and converting the voice search query to a text search query based on one or more probabilities, each of the probabilities corresponding to a word sequence in a group of one or more word sequences, the group including the first word sequence having the adjusted probability value.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit under 35U.S.C. § 120 of U.S. patent application Ser. No. 15/231,066, for“Dynamic Language Model,” which was filed on Aug. 8, 2016, now U.S. Pat.No. 10,140,362, which is a continuation of and claims the benefit under35 U.S.C. § 120 of U.S. patent application Ser. No. 15/006,392, whichwas filed on Jan. 26, 2016, now U.S. Pat. No. 9,418,143, which is acontinuation of and claims the benefit under 35 U.S.C. § 120 of U.S.patent application Ser. No. 14/719,178, which was filed on May 21, 2015,now U.S. Pat. No. 9,251,251, which is a continuation of and claims thebenefit under 35 U.S.C. § 120 of the filing date of U.S. patentapplication Ser. No. 13/802,414, which was filed on Mar. 13, 2013, nowU.S. Pat. No. 9,043,205, which claims the benefit under 35 U.S.C. §119(e) of U.S. Patent Application No. 61/662,889, which was filed onJun. 21, 2012. The foregoing applications are incorporated here byreference.

BACKGROUND

This specification relates to speech recognition.

Conventional voice search systems aim to convert a voice input from auser into a text search query. In a typical voice search system, a voiceinterface receives a user's voice input and provides the speech input toa speech recognition engine. The speech recognition engine can convertspoken words in the voice input to a text search query based on alanguage model. The voice search system then submits the text searchquery to a search engine. Thus, the voice search system allows a user toperform searches without typing the text search query.

SUMMARY

A voice search system applies a customized language model to convert aspeech input to a text search query. The voice search system obtains abase language model that is constructed, for example, using a collectionof anonymous search log data. The voice search system customizes thebase language model based on one or more customization criteria, e.g.,geographic location, time, or weather. In a customized language model,probability values assigned to one or more word sequences that arerelated to the customization criteria are adjusted.

When the voice search system receives a voice search query, the voicesearch system determines a context of the voice search query. The voicesearch system determines a customized language model when a component ofthe context satisfies a customization criterion. The voice search systemthen converts the voice search query into a text search query using thecustomized language model.

In general, one innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof receiving a base language model for speech recognition, the baselanguage model including a first word sequence having a base probabilityvalue; receiving a voice search query, the voice search query beingassociated with a query context; determining that a customized languagemodel is to be used when the query context satisfies one or morecriteria associated with the customized language model; obtaining thecustomized language model, the customized language model including thefirst word sequence having an adjusted probability value, the adjustedprobability value being the base probability value adjusted according tothe query context; and converting the voice search query to a textsearch query based on one or more probabilities, each of the one or moreprobabilities corresponding to a word sequence in a group of one or moreword sequences, the group of one or more word sequences including thefirst word sequence having the adjusted probability value. Otherembodiments of this aspect include corresponding computer systems,apparatus, and computer programs recorded on one or more computerstorage devices, each configured to perform the actions of the methods.A system of one or more computers can be configured to performparticular operations or actions by virtue of having software, firmware,hardware, or a combination of them installed on the system that inoperation causes or cause the system to perform the actions. One or morecomputer programs can be configured to perform particular operations oractions by virtue of including instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions.

The foregoing and other embodiments can each optionally include one ormore of the following features, alone or in combination. Obtaining thecustomized language model includes selecting the customized languagemodel from one or more customized language models based on the querycontext. Obtaining the customized language model includes adjusting thebase language model based on the received query context using one ormore language adjustment rules.

The query context includes a geographic location of a user device fromwhich the voice search query is received. The method includesdetermining that the word sequence relates to the geographic location,wherein the adjusted probability value includes the base probabilityvalue adjusted based on a degree of relationship between the wordsequence and the geographic location. The word sequence includes a nameof a feature located at or proximate to the geographic location; and thedegree of relationship is determined based on a distance between thefeature and the geographic location in the query context. The wordsequence includes a term of a past query received from the geographiclocation at a frequency that exceeds a threshold; and the degree ofrelationship is determined based on the frequency. The geographiclocation is a geographic area defined by one or more sets of latitudeand longitude coordinates. The query context includes a user providedidentifier for associating with the voice search query. The methodincludes determining a social group associated with the user providedidentifier, the social group including one or more user identifiers thatare related to the user provided identifier; determining a frequency ofoccurrence of the word sequence using stored query logs that areassociated with the user identifiers in the social group; and adjustingthe base probability value based on the frequency of occurrence.Adjusting the base probability value is further based on a degree ofrelationship in the social group between each of the user identifiersand the user provided identifier. The query context includes a timestampof the voice search query. Obtaining the customized language modelincludes: identifying one or more electronic documents, the one or moreelectronic documents being associated with a number of search queriesaccording to a query log, the number of search queries satisfying athreshold number, the search queries having occurred within a specifiedtime window of the timestamp, each of the one or more electronicdocuments including the word sequence; and adjusting the baseprobability value based on the number of search queries. Each of the oneor more electronic documents is a news article. The method includesproviding for display on a display device, as a response to the voicesearch query, one or more search results of the text search query.Converting the voice search query to a text search query comprisesconverting the voice search query to a text search query that includesthe first word sequence based on the corresponding adjusted probabilityvalue in the customized language model.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. Compared to a conventional voice search system, avoice search system using a dynamic language model can provide morepertinent text search queries based on a received voice input. Since thesystem takes consideration of a particular circumstance of the voiceinput in a speech-to-text conversion, including, for example, a time ofreceiving the voice input, the location where the voice input is issued,and/or a user profile of a user providing the voice input, the resultingquery can be customized according to the particular circumstance. Thus,the system can provide enhanced user experience.

The details of one or more embodiments of dynamic language modelingtechniques are set forth in the accompanying drawings and thedescription below. Other features, objects, and advantages of thetechniques will be apparent from the description and drawings, and fromthe claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram providing an overview of an example system forproviding voice searching.

FIG. 2 is a block diagram illustrating example rules for generating adynamic language model.

FIG. 3 is a diagram illustrating example geography-based rules of adynamic language model.

FIG. 4 is a diagram illustrating example social group based rules of adynamic language model.

FIGS. 5A and 5B are block diagrams illustrating example implementationsof dynamic language models.

FIG. 6 is a flowchart of an example method for using a customizedlanguage model.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

FIG. 1 is a block diagram providing an overview of an example system forproviding voice searching. A voice search system 102 includes one ormore computers programmed to receive a user's speech input from a user,convert the speech input into a text search query using a dynamiclanguage model, receive search results responsive to the text searchquery, and provide the search results to the user in response to thespeech input. The voice search system 102 can be implemented, forexample, on one or more computers including a server or on a userdevice.

The voice search system 102 includes a voice interface 104 configured toreceive a speech input 106 from a user. The voice interface 104 caninclude, for example, an acoustic-to-electric transducer or sensor(e.g., a microphone) coupled to a user device. The speech input 106 caninclude a sequence of spoken words.

The voice search system 102 includes a speech recognition subsystem 108.The speech recognition subsystem 108 can be a software component of thevoice search system 102 configured to convert the spoken words in thespeech input 106 into a text search query 115. The speech recognitionsubsystem 108 includes a base language model 110. The base languagemodel 110 can include probability values associated with sequences ofwords. In the base language model 110, a given word sequence, e.g., (Wa,Wb, . . . , Wm) is associated with a probability value, e.g., P(Wa, Wb,. . . , Wm), where each of Wa, Wb, . . . , Wm includes a word, phrase,or sentence. The base language model 110 can include multiple wordsequences and a probability value associated with each of the wordsequences.

The base language model 110 can be an N-gram model (e.g., a unigrammodel or a multigram model). In the example shown, the base languagemodel 110 is a bigram language model. The base language model includes aprobability value P(Wx, Wy) associated with a word sequence (Wx, Wy).The probability value P(Wx, Wy) can be calculated using the followingformula:P(Wx,Wy)≈P(Wx|<s>)P(Wy|Wx),  (1)where P(Wx, Wy) is a probability of the word sequence (Wx, Wy) appearingin a search query, <s> indicates a start of a search query, P(Wx|<s>)indicates a conditional probability that, given <s>, word Wx follows<s>, and P(Wy|Wx) indicates a conditional probability that, given wordWx, word Wy follows word Wx.

In addition, the base language model 110 can include a probability valueP(Wp, Wq, Wr) that is associated with the word sequence (Wp, Wq, Wr).The probability value P(Wp, Wq, Wr) can be calculated, for example, in abigram model using the following formula:P(Wp,Wq,Wr)≈P(Wp|<s>)P(Wq|Wp)P(Wr|Wq)  (2)The probability values P(Wx|<s>), P(Wy|Wx), P(Wp|<s>), P(Wq|Wp), andP(Wr|Wq) can be determined from stored queries received in the past.

For illustration, the word sequence (Wp, Wq, Wr) can represent a searchquery that is popular among the general public, e.g., a search querythat appears more frequently compared to other search queries. The wordsequence (Wx, Wy) can represent a search query that is popular onlyunder special circumstances. For example, (Wx, Wy) can be a name of aspecialty restaurant located in a geographic area, e.g., a town, ABC.The voice search system 102 is more likely to receive a search queryhaving query terms (Wx, Wy) from users who are in geographic area ABCthan from users who are out of the geographic area ABC.

In this example, geographic area ABC is not a particularly populargeographic area among many geographic areas from which queries arereceived. Thus, in the base language model, P(Wp, Wq, Wr) can be greaterthan P(Wx, Wy), indicating that, in general, the speech recognitionsubsystem 108 is more likely to convert the speech input 106 into a textsearch query (Wp, Wq, Wr) than into a text search query (Wx, Wy).According to a ranking of probability values, (Wp, Wq, Wr) can be rankedas first, while (Wx, Wy) can be ranked at some nth position after thefirst in the base language model 110.

The speech recognition subsystem 108 includes a model customizationmodule 112. The model customization module 112 can be a softwarecomponent of the speech recognition subsystem 108 configured to cause acomputing device to adjust the base language model 110 dynamically basedon one or more adjustment factors and to generate a customized languagemodel 114. The adjustment factors are information items separate fromthe base language model 110 that can affect the likelihood that thespeech input 106 is converted to a particular text search query. Themodel customization module 112 can determine the adjustment factorsbased on, for example, one or more of location, query timestamp, orother parameters.

For example, for a given speech input, the probability value P(Wx, Wy)can be lower than the probability value P(Wp, Wq, Wr) in the baselanguage model 110. The model customization module 112 can increase theprobability value P(Wx, Wy) such that the probability value P(Wx, Wy) inthe customized language model 114 is higher than the probability valueP(Wx, Wy) in the base language model 110, when one or more adjustmentfactors are taken into consideration, for example when a particularlocation condition is satisfied. The model customization module 112 canassociate the customized language model 114 with a location (e.g.,geographic region ABC). The model customization module 112 candynamically adjust the base language model 110 to generate thecustomized language model 114 at query time such that the probability ofone or more particular word sequences in the customized language model114 is different from the probability of those same word sequences inthe base language model 110.

The speech recognition subsystem 108 is configured to receive a querycontext 117 that is associated with the speech input 106. The querycontext 117 includes data that describe a circumstance under which thespeech input 106 is generated. The query context 117 can includelocation information indicting where the speech input 106 is issued. Forexample, the location information can include a longitude and a latitudeparameter, global positioning system coordinates, or an internetprotocol (IP) address based on which a location can be estimated. Thelocation information can be obtained from a user device.

The speech recognition subsystem 108 is configured to determine whetherto use the base language model 110 or the customized language model 114to convert the speech input 106 to a text search query 115. If, forexample, the speech recognition subsystem 108 determines, based on thequery context 117, that a user providing the speech input 106 is locatedin geographic region ABC, the speech recognition subsystem 108 can usethe customized language model 114. However, if the speech recognitionsubsystem 108 determines, based on the query context 115, that a userproviding the speech input 106 is not located in geographic region ABC,the speech recognition subsystem 108 can use the base language model 110to convert the speech input 106 to the text search query 115.

Thus, if the speech recognition subsystem 108 determines that the userproviding the speech input 106 is located in geographic region ABC, thespeech recognition subsystem 108 applies the customized language model114 in speech recognition, and generates a text string from the speechinput 106. According to a ranking of probability values in thecustomized language model 114, everything else being equal, the speechinput 106 is more likely to relate to (Wx, Wy) than to (Wp, Wq, Wr),because the increased probability value P(Wx, Wy) in the customizedlanguage model 114 is higher than the probability value P(Wp, Wq, Wr).Accordingly, the speech recognition subsystem 108 converts the speechinput 106 into a text string including terms (Wx, Wy) or provides fordisplay a list of options in which (Wx, Wy) is placed higher than (Wp,Wq, Wr), e.g., at or near the top of a query suggestion list, for a userto select a text search query.

If the speech recognition subsystem 108 determines that the userproviding the speech input 106 is not located in a particular geographicregion associated with a customized language model, the speechrecognition subsystem 108 applies the base language model 110 in speechrecognition, and generates a text string from the speech input 106,e.g., that includes the terms (Wp, Wq, Wr).

The voice search system 102 submits the text string generated by thespeech recognition subsystem 108 as a text search query 115 to a searchengine 116. The text search query 115 can include search terms “Wx Wy”based on the customized language model 114 or “Wp Wq Wr” based on thebase language model 110. The search engine 116 can include a web searchengine coupled to the voice search system 102. The voice search system102 receives one or more query results 118 from the search engine 116,and provides the query results 118 for display on a presentationinterface 120, which can be a component on a user device, e.g., a webbrowser, or can be a component of a server, e.g., a dynamic web page.The presentation interface 120 can include a display interface or avoice output interface. The presentation interface 120 can present tothe user the one or more query results 118, e.g., a displayed queryresult page 122, which can include, for example, a result “Wx WyRestaurant.”

FIG. 2 is a block diagram 200 illustrating example rules for generatinga customized language model, e.g., customized language model 114. Asystem implementing a dynamic language model obtains a base languagemodel 202. The system can obtain the base language model 202 fromanother system. The system can alternatively build the base languagemodel 202 from search logs 204 using publicly available languagemodeling technologies, e.g., RandLM or the CMU-Cambridge StatisticalLanguage Modeling Toolkit. The system can obtain the search logs from asearch engine that is in communication with the system or that is a partof the system. In some implementations, the system obtains a separatebase language model 202 for each language supported by the system.

The system can customize the base language model 202 using geographiclanguage model rules 206. The geographic language model rules 206include adjustment factors for adjusting a probability value of a wordsequence in the base language model 202 when the word sequence relatesto a geographic feature in, or is received from, a specified geographicarea. The word sequence can relate to the geographic feature by virtueof being, e.g., a name, an address, or a characteristic of thegeographic feature.

The geographic feature can include any feature that relates to, orentity that is located at, the geographic area. For example, geographicfeatures can include businesses, landmarks, local government offices,streets, parks, schools, or events commonly occur at a particularlocation. The geographic areas can include countries, states, counties,cities, business districts, or grids determined by latitudes andlongitudes.

The geographic language model rules 206 can be grouped by geographicareas. For each geographic area, the system associates word sequencesthat relate to the geographic features in the geographic area with anadjustment factor. The adjustment factor can indicate a likelihood that,compared with other geographic areas, a search query from the specifiedgeographic area is more likely to include a word sequence that relatesto a geographic feature in a specified geographic area. The adjustmentfactor can include a multiplier that is applied to the probability valueassociated with the word sequence in the base language model 202.

For example, a geographic language model rule includes an adjustmentfactor for a word sequence (Wx, Wy), which, in this example, is a nameof a business, located in geographic region ABC. The adjustment factorcan include a multiplier that increases the probability value P(Wx, Wy)of the word sequence (Wx, Wy). The system applies the multiplier to theprobability value P(Wx, Wy) in the base language model 202 to create acustomized language model. The customized language model can beassociated with geographic region ABC. The system uses the customizedlanguage model for speech recognition when the system determines that auser providing the speech input (e.g., a voice query) is located ingeographic region ABC.

The system can generate the geographic language model rules 206 based oninformation from a geographic data store 208, search logs 204, or both.The geographic data store 208 stores names and locations of geographicfeatures associated with each geographic area. The system can obtain thenames and locations from a list of registered business names, publiclyrecords, or user inputs. For example, an operator of a restaurant havingname (Wx, Wy) in geographic region ABC can register the name (Wx, Wy) inassociation with geographic region ABC in a web “yellow page” or otherdatabase of public business listings in the geographic data store 208.Based on the database, the system can associate the name (Wx, Wy) withgeographic region ABC. The names stored in the geographic data store 208can be normalized (e.g., de-capitalized, spell corrected, or otherwisestandardized) to map to a word sequence in the base language model 202.

The system can analyze search logs 204 to generate query frequencyinformation that the system uses to determine a magnitude of anadjustment factor. The query frequency information can include a counton the number of times a particular query has been received by thesystem or by the search engine. The system can filter the queryfrequency information using a threshold. The system can use thethreshold to determine whether to increase a probability value. Usinginformation stored in the search logs 204, the system can determine afrequency of a voice or text search query that includes query terms (Wx,Wy) received from users at various locations. If the system determinesthat the frequency exceeds a threshold when the users are located at ornear geographic region ABC, the system can specify a geographic languagemodel rule having an adjustment factor that increases the value P(Wx Wy)in the base language model 202.

The system can additionally or alternatively customize the base languagemodel 202 using social group language model rules 210. The social grouplanguage model rules 210 include adjustment factors for adjusting aprobability value in the base language model 202 based on a user'ssocial group. The probability value for a word sequence in the baselanguage model 202 is adjusted for user A based on information submittedby users in the social group of user A. The information can include, forexample, one or more stored queries of the users in the social group.

If the system receives a particular query from users among user A'ssocial group at a frequency that exceeds a threshold, the system canadjust the probability value of a word sequence that corresponds to thequery. For example, the system can determine that, based on queries fromusers within user A's social group, the system receives a thresholdnumber of search queries including word sequence (Wa, Wb, Wc, Wd). Insome implementations, queries have been anonymized in query logs suchthat the submitting user is obscured (e.g., by removing sourceidentifying information from stored queries). Based on thedetermination, the system can adjust the base language model 202 toincrease a probability value for the word sequence (Wa, Wb, Wc, Wd) foruser A, as well as for users within the social group of user A. Thus,for users within the social group of user A, the system can increase thelikelihood of recognizing a speech input as word sequence (Wa, Wb, Wc,Wd) rather than some other word sequence, e.g., (We, Wf, Wg, Wh), havinga higher probability in the base language model 202.

The system can generate the social group language model rules 210 basedon information obtained from a social group data store 212, search logs204, or both. The social group data store 212 can store information thatrelates users and that each of the related users choose to submit to thesystem for customizing language models. The information can include acircle of connected users, which include one or more users identified associally connected to the user. The information can be retrieved from asocial site, a virtual group, a virtual chat room, a contact list, orany combination of the above. The search logs 204 stores query frequencyinformation, which can include a count on the number of times aparticular query has been received by the system and correspond to useridentifiers in a user's social group.

The system can additionally or alternatively customize the base languagemodel 202 using event language model rules 214. The event language modelrules 214 include adjustment factors for adjusting a probability valuein the base language model 202. When a word sequence is associated witha popular event, the probability value associated with the word sequencecan be increased. The system generates the event language model rules214 based on information stored in the event data store 216, or thesearch logs 204, or both. The specified events can include any eventthat can affect the probability that a speech input converts into aparticular word sequence.

In some implementations, the system adjusts probability values based ontime events. The system can generate the event language model rules 214based on time of day or day of week. For example, the system candetermine that in specified hours of a day (e.g., between noon and 1 pm,or between 6 pm and 8 pm), the system receives a number or percentage ofqueries that relate to restaurants, where the number or percentageexceeds a threshold. The system can increase the probability values ofword sequence in the base language model 202 that relate to restaurantsbased on the number or percentage.

In some implementations, the system adjusts probability values based onweather events. The system can generate the event language model rules214 based on weather information. The system can be configured such thatunder given weather conditions, a specified set of word sequences aregiven adjusted probability values. The event data store 216 can includeinformation received from one or more weather services. When each of theweather services provides a weather forecast that a probability of rainexceeds a specified threshold, names of services relating to roadcondition reports, flood warning, or roof repair can be given higherweight.

In some implementations, the system uses popular news events. The systemcan generate a particular event language model rules 214 based on apopularity of subjects in one or more news reports. For example, theevent data store 216 can store information from various news sources.When a word sequence appear in the news sources at a frequency thatexceeds a threshold, the system can generate an event language modelrule 214 that increases the probability value associated with the wordsequence.

FIG. 3 is a diagram illustrating example dynamic language models basedon geographic information. In a system implementing a dynamic languagemodel based on geographic information, the system annotates wordsequences with the geographic information. At query time, the systemuses the dynamic language model based on a geographic location of a userdevice from which a speech input is received.

The system includes a location interface configured to receive locationinformation from the user device. In some implementations, the locationinterface can include various modules configured to receive latitude,longitude, or altitude information from a location-aware user device. Insome implementations, the location interface can include various modulesthat are configured to determine a location of the user device based onan Internet protocol (IP) address of the user device, a wireless accesspoint to which the user device is connected, or other information. Thesystem can convert a same speech input to different text search querieswhen the user device is located at different geographic locations.

The system divides a geographic area into a collection of virtual blocks310. Each block in the collection of virtual blocks 310 can correspondto a language model customized for the block. The blocks can be definedbased on latitude and longitude coordinates of a center of each block orone or more vertices of each block. In some implementations, the blockscan be defined based on a corner of each block, and a block size. Theblock size can be defined in kilometers, miles, or degrees of latitudeor longitude.

The system identifies an entity 312 located in Block 7 of the collectionof virtual blocks 310. The entity 312 can be a feature that ispermanently or quasi-permanently associated with Block 7. For example,the entity 312 can be a business located in block 7, the business havinga business name (e.g., word sequence (Wx, Wy)) registered in the systemor registered in a registry coupled to the system. The system can createa dynamic language model associated with Block 7. In this dynamiclanguage model, the probability value P(Wx, Wy) of the word sequenceassociated with the entity is adjusted by a multiplier, such that theprobability value is increased.

The system can create a customized language model for each block of thecollection of virtual blocks 310. In each customized language model, theprobability value of the word sequence associated with the entity 312 isadjusted by an amount that is determined based on a distance betweenBlock 7 and the block to which the dynamic language model is associated.In particular, the amount can be discounted based on the distance. Thedistance can be, for example, a Euclidean distance, a Manhattandistance, or a distance measured by estimated travel time. For thoseblocks that are immediate neighbors of Block 7 (e.g., Blocks 4, 5, and8), the corresponding dynamic language models can have a firstadjustment amount for the probability value P(Wx, Wy). For those blocksthat are distant neighbors of Block 7 (e.g., Blocks 1-3, 6, and 9), thecorresponding dynamic language models can have a second adjustmentamount for the probability value P(Wx, Wy), where the second adjustmentamount is less than the first adjustment amount. For those blocks thatare farther away from Block 7 (not shown), the probability value P(Wx,Wy) can remain unadjusted.

When the system receives a speech input from a user device, the systemdetermines in which of the collection of virtual blocks 310 the userdevice is located. The system then applies the language model associatedwith the block to convert the speech input into a text search query.

FIG. 4 is a diagram illustrating an example dynamic language model 400based on a user's social group. For a given user, the system adjustsprobability values of word sequence in a language model when the wordsequence is used by users in the user's social group at a frequency thatsatisfies a threshold. The frequency can be an absolute frequency or arelative frequency.

The system can receive user data 402 of a registered user (user A) of aparticular social site. User data 402 can include a user identifier foruser A and user information submitted by user A after user A requestsspeech recognition improvement. The user information can include, forexample, a user's home location, a user preference, and otherinformation associated with the user.

The system generates language model adjustment rules 404 for user Abased on the user data 402. The system can generate the language modeladjustment rules 404 based on the location, preference, and otherinformation. For example, if user A has submitted a query including aparticular query term, and if user A has submitted that query for aspecified number of times (e.g., once, or more than once), the systemcan create a rule in language model adjustment rules 404 that increasesthe probability of a word sequence including the query term.

The system can identify a social network of user A based on the userdata 402 as provided or authorized by the user. The social network caninclude a user-specified social connection circle (e.g., a friendcircle), a list of users based on user A's contact list, or a list ofusers based on common activities between the users and user A (e.g.,chat room participation). The system can determine a degree ofrelationship between the users and user A. For example, the system candetermine that user B and user C are directly related to user A and thusare related to user A at a first degree. The system can determine thatusers D, E, F, and G are related to user A through users B and C, andthus are related to user A at a second degree.

The system receives user data 412, 414, 416, 418, 420, and 422 fromusers in user A's social network (user B through user G), after theseusers request to improve speech recognition for themselves and users intheir social connection circle. The system can create model adjustmentrules 432, 434, 436, 438, 440, and 442 for user B through user G,respectively. The system can generate at least a portion of the languagemodel adjustment rules 404 based on the user data 412-422, or the modeladjustment rules 432-442, or both.

In some implementations, the system uses queries of a user's socialconnections to build a customized language model. For example, accordingto user data 412, user B has performed a search using a query containingcertain query terms one or more times. The system increases aprobability value of a word sequence including the query terms based ona frequency of user B's search and a degree of relationship between userB and user A.

In some implementations, the system uses information other than queriesof a user's friends to build a customized language model. If, forexample, user B and user C have discussed a restaurant having a name “WxWy” with or without user A's participation, or have searched using aquery “Wx Wy,” the system can increase the probability for the wordsequence (Wx, Wy) in a customized language model for user A.

The application of user A's social network can be based on privacysettings of each user. For example, the system can use user B and userC's search to adjust the language model of user A if each of user B anduser C gives permission to share search information among a circle offriends, or if each of user B and user C gives permission to use theuser's information to tune language models of friends.

The system can determine the amount of adjustment based on the degree ofrelationship. A lower degree of relationship represents a closerrelationship between two users and can indicate a higher likelihood thatthe two users have shared interests and search habits. Accordingly, thesystem decreases the amount of adjustment as the degree of relationshipincreases. For example, the system can increase the probability valuefor a word sequence for user A at a first amount if the increase isbased on query performed by user B. The system than can increase theprobability value for a word sequence for user A at a second amount ifthe increase is based on query performed by user D. The first amount canbe greater than the second amount.

The system can customize a base language model using the identifiedlanguage model adjustment rules 404 at query time, or store a customizedlanguage model before query time. When the system receives a speechinput from user A, the system can identify the language model adjustmentrules 404 for user A based on an identifier of user A, and customize abase language model using the language model adjustment rules 404.Alternatively or additionally, the system can perform speech recognitionusing a pre-stored custom language model that is specific to user A.

FIGS. 5A and 5B are block diagrams illustrating example implementationsof dynamic language models. A system implementing dynamic language modeltechniques can adopt implementations described in reference to FIG. 5A,FIG. 5B, or a combination of both.

FIG. 5A is a block diagram 500 illustrating example implementationswhere the system creates customized language models 502, 504, 506, and508 asynchronous with a voice query, e.g., at some time other than inresponse to a received query. Each of the customized language models502, 504, 506, and 508 includes some or all of the word sequences in abase language model, and is associated with one or more criteria forapplying the customized language model. The probability valuesassociated with the respective word sequences in a customized languagemodel are determined based on the application criteria. Each of theapplication criteria can specify a location, time, weather, or otherfactors (e.g., user segments defined by age, gender, income level, orbuying habit of the general population). For example, the dynamiclanguage model 506 is associated with application criteria specifyingthat the dynamic language model 506 applies at a specified location in aspecified time window under a specified weather condition. The systemcan build or update the customized language models 502, 504, 506, and508 periodically or upon a triggering event. The triggering event caninclude, for example, a new way of segmenting users has beenimplemented.

At query time, the system receives a speech input and obtains a contextof the speech input. For example, the system can determine the user'slocation, time of the speech input, and retrieve weather informationfrom a weather service based on the location and time. The system canidentify a user segment that the user belongs to using a profileassociated with a user identifier provided by the user when the userlogs into the system. The system submits the context to a language modelselector 510. The language model selector 510 is a component of thesystem configured to match the context of the speech input with theapplication criteria associated with the customized language models 502,504, 506, or 508, and identify a dynamic language model based on thematch.

For example, the system can receive a user location, a query time, andquery time weather at the user location. The language model selector 510can determine that the user location intersects the applicable locationof the application criteria associated with the dynamic language model506, the query time is within the applicable time window of the of theapplication criteria, and the query time weather at the user locationmatches the applicable weather of the application criteria. Accordingly,the language model selector 510 selects customized language model 506.The system then performs speech recognition to convert the user's speechinput into a text search query using the customized language model 506.If, for example, the system receives the same speech input at adifferent location or at a different time, the system can apply adifferent language model (e.g., language model 504).

FIG. 5B is a block diagram 520 illustrating example implementationswhere the system dynamically adjusts a language model 522. The languagemodel 522 can be a base language model or a customized language model.

The system includes a language model modifier 524. The language modelmodifier is a component of a user device or a server device configuredto apply one or more rules 526 to adjust the language model 522. Thelanguage model modifier 524 receives inputs including personalizedinformation, location information, date/time information, weatherinformation, recent news/current affairs information, or query streamfrom one or more users. The language model modifier 524 applies therules 526 to the language model 522 based on the input. Afterapplication of the rules 526, the language model 522 can become acustomized language model.

For example, one of the rules 526 can specify that when a term appearsin news reports within a specified period of time (e.g., last three daysbefore the rules 526 are applied to the language model 522) and at afrequency that satisfies a threshold, the system increases theprobability value of a word sequence including that term. Another rulecan specify that a user can allow the system to analyze queriessubmitted by the user in the past and determine a search pattern to beassociated with the user, if the user chooses to allow the system tocollect the information.

In some implementations, the language model modifier 524 can adjust thelanguage model 522 at query time. In some implementations, the languagemodel modifier 524 can create a customized language model for aparticular user or in response to a particular current event, and storethe customized language model on a user device or a server device.

FIG. 6 is a flowchart of an example method 600 for using a customizedlanguage model. For convenience, the method will be described withrespect to a system that performs the method 600, e.g., as describedabove. The system receives (602) a base language model for speechrecognition. The base language model includes multiple word sequences,each word sequence being associated with a probability value. The wordsequences can be represented as text string, compressed text strings,objects, or indices. The base language model can be generated fromanonymous query logs of a collection of users.

The system receives (604) a voice search query. The voice search queryis associated with a query context. The query context can include ageographic location of a user device from which the voice search queryis received. The geographic location can be a point or a geographicarea. The point or geographic area can be defined, for example, by oneor more sets of latitude, longitude, or altitude coordinates. The querycontext can include, for example, a timestamp of the query, or a useridentifier of a user providing the voice search query.

The system determines (605) that a customized language model is to beused when the query context satisfies one or more criteria associatedwith the customized language model. If the query context does not meetany criterion, the system can use the base language model to convert thevoice search query to a text search query.

The system obtains (606) the customized language model. The customizedlanguage model includes one or more word sequences corresponding to anadjusted probability value. The adjusted probability value is the baseprobability value adjusted according to the query context, when thequery context matches one or more adjustment criteria. Obtaining thecustomized language model can include at least one of: (1) selecting thecustomized language model from one or more pre-stored customizedlanguage models based on the query context, or (2) adjusting the baselanguage model at query time based on the received query context usingone or more language adjustment rules to generate the customizedlanguage model. The customized language model can be obtained asdescribed above with respect to FIGS. 1-5

In some implementations, the query context includes a geographiclocation of a user device from which the voice search query is received.The system can identify one or more word sequences related to thegeographic location. The adjusted probability value of a word sequenceincludes the base probability value adjusted based on a degree ofrelationship between the word sequence and the geographic location. Forexample, the word sequence can include a name of a feature (e.g.,business, park, or street) located at or proximate to the geographiclocation of the user device. The degree of relationship can bedetermined based on a distance between the feature and the geographiclocation in the query context. The geographic location can be ageographic area defined by one or more sets of latitude, longitude, oraltitude coordinates.

In some implementations, a word sequence includes a term of a past queryreceived from the geographic location at a frequency that exceeds athreshold. The degree of relationship is determined based on thefrequency. For example, if the system receives a sufficient number orpercentage of queries including terms “climate best by government test”at a particular location, the system can increase a probability valuefor the word sequence “climate best by government test” in a customizedlanguage model associated with that particular location. In someimplementations, the higher the number or percentage, the higher theincrease the probability value.

In some implementations, the query context includes a user providedidentifier (e.g., a user name or a user identifier) for associating withthe voice search query. The system determines a social group associatedwith the user provided identifier. The social group can be a storedsocial network of a user. The social group can include one or more useridentifiers that are related to the user provided identifier in thesocial network. The system determines a frequency of occurrence of theword sequence using stored query logs that are associated with the useridentifiers in the social group, and adjusts the base probability valuebased on the frequency of occurrence. For example, the system candetermine that a search query, either voice or text, that includessearch terms “swim meet” occurs in a high frequency in a group relatingto parenting. The system can create a customized language model in whichthe probability value for word sequence “swim meet” is higher than thatin a base language model, and apply the customized language model tospeech input by a user in the parenting group. The system canadditionally or alternatively adjust the base probability value based ona degree of relationship in the social group.

In some implementations, the query context includes a timestamp of thevoice search query. Obtaining the customized language model includesidentifying one or more electronic documents (e.g., news articles). Theelectronic documents, each individually or as a collection, areassociated with a number of search queries according to a query log. Anelectronic document can be associated with a query if it has beendesignated as a search result of the query by a search engine orselected as a destination from a list of search results by a user. Thesystem identifies the electric documents that each includes one or moreword sequences. If the associated search queries occur within aspecified time window (e.g., a day or a week) from the timestamp, thesystem can increase the probability values associated with the one ormore word sequences in a customized language model. For example, if, fora period of time, numerous education related news articles include aword sequence “tiger mom,” the word sequence “tiger mom,” which may beassociated with a low probability value in a base language model, can beassociated with a higher probability value in a customized languagemodel.

The system converts (608) the voice search query to a text search querythat includes the word sequence, based on the corresponding adjustedprobability value in the customized language model. The system canprovide (610) one or more search results of the text search query as aresponse to the voice search query for display on a display device.

Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage media for execution by, or tocontrol the operation of, a data processing apparatus. Alternatively orin addition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal, that is generated toencode information for transmission to a suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially-generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate physical components or media, e.g., multiple CDs, disks, orother storage devices.

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processing unit, a computer, a system on a chip, a personalcomputer system, desktop computer, laptop, notebook, netbook computer,mainframe computer system, handheld computer, workstation, networkcomputer, application server, storage device, a consumer electronicsdevice such as a camera, camcorder, set top box, mobile device, videogame console, handheld video game device, a peripheral device such as aswitch, modem, router, or in general any type of computing or electronicdevice, or multiple ones, or combinations, of the foregoing. Theapparatus can include special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application-specificintegrated circuit). The apparatus can also include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, across-platform runtime environment, a virtual machine, or a combinationof one or more of them. The apparatus and execution environment canrealize various different computing model infrastructures, such as webservices, distributed computing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram can, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processing units executing one ormore computer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processing units suitable for the execution of a computer programinclude, by way of example, both general and special purposemicroprocessors, and any one or more processing units of any kind ofdigital computer. Generally, a processing unit will receive instructionsand data from a read-only memory or a random access memory or both. Theessential elements of a computer are a processing unit for performingactions in accordance with instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto-optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio or video player, a game console, a GlobalPositioning System (GPS) receiver, a network routing device, or aportable storage device (e.g., a universal serial bus (USB) flashdrive), to name just a few. Devices suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including by way of example semiconductormemory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto-opticaldisks; and CD-ROM and DVD-ROM disks. The processing unit and the memorycan be supplemented by, or incorporated in, special purpose logiccircuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or a routing device, e.g., anetwork router, or any combination of one or more such back-end,middleware, or front-end components. The components of the system can beinterconnected by any form or medium of digital data communication,e.g., a communication network. Examples of communication networksinclude a local area network (“LAN”) and a wide area network (“WAN”), aninter-network (e.g., the Internet), and peer-to-peer networks (e.g., adhoc peer-to-peer networks).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs executing on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data (e.g., an HTML page) to a clientdevice (e.g., for purposes of displaying data to and receiving userinput from a user interacting with the client device). Data generated atthe client device (e.g., a result of the user interaction) can bereceived from the client device at the server.

A system of one or more computers can be configured to performparticular actions by virtue of having software, firmware, hardware, ora combination of them installed on the system that in operation causesor cause the system to perform the actions. One or more computerprograms can be configured to perform particular actions by virtue ofincluding instructions that, when executed by data processing apparatus,cause the apparatus to perform the actions.

An electronic document (which for brevity will simply be referred to asa document) can, but need not, correspond to a file. A document can bestored in a portion of a file that holds other documents, in a singlefile dedicated to the document in question, or in multiple coordinatedfiles.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what can be claimed, but rather as descriptions offeatures specific to particular embodiments of particular inventions.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features can be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination can be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingcan be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing can be advantageous. Accordingly, other embodiments arewithin the scope of the following claims.

What is claimed is:
 1. A method comprising: receiving a voice searchquery, the voice search query provided by a user to a user device;determining that a particular customized language model of a pluralityof customized language models is to be used to recognize the voicesearch query, wherein each customized language model includes one ormore adjusted probabilities for respective word sequences of thelanguage model, wherein the probabilities are adjusted with respect tocorresponding word sequence probabilities in a base language model;using the particular customized language model to generate a text searchquery from the voice search query; and providing for display on the userdevice, one or more search results responsive to the text search query.2. The method of claim 1, wherein each customized language model isgenerated according one or more language adjustment rules for adjustingthe respective probabilities of the base language model based on aparticular geographic location.
 3. The method of claim 1, whereindetermining that the particular customized language model is to be usedcomprises determining that the query is received by a user device havinga particular geographic location associated with the particularcustomized language model.
 4. The method of claim 3, wherein adjusting aprobability for the particular customized language model includesadjusting a probability of a particular word sequence based on a degreeof relationship between the particular word sequence and a geographiclocation of the user device.
 5. The method of claim 4, wherein: theparticular word sequence includes a name of a feature located at orproximate to the geographic location; and the degree of relationship isdetermined based on a distance between the feature and the geographiclocation in the query context.
 6. The method of claim 1, wherein thedetermining that the particular customized language model is to be usedcomprises determining one or more terms of the query and occurrences ofthe one or more terms by users sharing a social group with the user. 7.The method of claim 6, wherein adjusting a probability for theparticular customized language model includes determining a frequency ofoccurrence of the word sequence using stored query logs that areassociated with users of the social group and adjusting a probability ofa particular word sequence based on the frequency.
 8. The method ofclaim 1, wherein the determining that the particular customized languagemodel is to be used comprises determining an association between thereceived query and one or more current events.
 9. A system comprising:one or more computers and one or more storage devices storinginstructions that are operable, when executed by the one or morecomputers, to cause the one or more computers to perform operationscomprising: receiving a voice search query, the voice search queryprovided by a user to a user device; determining that a particularcustomized language model of a plurality of customized language modelsis to be used to recognize the voice search query, wherein eachcustomized language model includes one or more adjusted probabilitiesfor respective word sequences of the language model, wherein theprobabilities are adjusted with respect to corresponding word sequenceprobabilities in a base language model; using the particular customizedlanguage model to generate a text search query from the voice searchquery; and providing for display on the user device, one or more searchresults responsive to the text search query.
 10. The system of claim 9,wherein each customized language model is generated according one ormore language adjustment rules for adjusting the respectiveprobabilities of the base language model based on a particulargeographic location.
 11. The system of claim 9, wherein determining thatthe particular customized language model is to be used comprisesdetermining that the query is received by a user device having aparticular geographic location associated with the particular customizedlanguage model.
 12. The system of claim 11, wherein adjusting aprobability for the particular customized language model includesadjusting a probability of a particular word sequence based on a degreeof relationship between the particular word sequence and a geographiclocation of the user device.
 13. The system of claim 12, wherein: theparticular word sequence includes a name of a feature located at orproximate to the geographic location; and the degree of relationship isdetermined based on a distance between the feature and the geographiclocation in the query context.
 14. The system of claim 9, wherein thedetermining that the particular customized language model is to be usedcomprises determining one or more terms of the query and occurrences ofthe one or more terms by users sharing a social group with the user. 15.The system of claim 14, wherein adjusting a probability for theparticular customized language model includes determining a frequency ofoccurrence of the word sequence using stored query logs that areassociated with users of the social group and adjusting a probability ofa particular word sequence based on the frequency.
 16. The system ofclaim 9, wherein the determining that the particular customized languagemodel is to be used comprises determining an association between thereceived query and one or more current events.
 17. One or morenon-transitory computer storage media encoded with a computer program,the program comprising instructions that are operable, when executed bydata processing apparatus, to cause the data processing apparatus toperform operations comprising: receiving a voice search query, the voicesearch query provided by a user to a user device; determining that aparticular customized language model of a plurality of customizedlanguage models is to be used to recognize the voice search query,wherein each customized language model includes one or more adjustedprobabilities for respective word sequences of the language model,wherein the probabilities are adjusted with respect to correspondingword sequence probabilities in a base language model; using theparticular customized language model to generate a text search queryfrom the voice search query; and providing for display on the userdevice, one or more search results responsive to the text search query.18. The one or more non-transitory computer storage media of claim 17,wherein each customized language model is generated according one ormore language adjustment rules for adjusting the respectiveprobabilities of the base language model based on a particulargeographic location.
 19. The one or more non-transitory computer storagemedia of claim 17, wherein determining that the particular customizedlanguage model is to be used comprises determining that the query isreceived by a user device having a particular geographic locationassociated with the particular customized language model.
 20. The one ormore non-transitory computer storage media of claim 19, whereinadjusting a probability for the particular customized language modelincludes adjusting a probability of a particular word sequence based ona degree of relationship between the particular word sequence and ageographic location of the user device.
 21. The one or morenon-transitory computer storage media of claim 20, wherein: theparticular word sequence includes a name of a feature located at orproximate to the geographic location; and the degree of relationship isdetermined based on a distance between the feature and the geographiclocation in the query context.
 22. The one or more non-transitorycomputer storage media of claim 17, wherein the determining that theparticular customized language model is to be used comprises determiningone or more terms of the query and occurrences of the one or more termsby users sharing a social group with the user.
 23. The one or morenon-transitory computer storage media of claim 22, wherein adjusting aprobability for the particular customized language model includesdetermining a frequency of occurrence of the word sequence using storedquery logs that are associated with users of the social group andadjusting a probability of a particular word sequence based on thefrequency.
 24. The one or more non-transitory computer storage media ofclaim 17, wherein the determining that the particular customizedlanguage model is to be used comprises determining an associationbetween the received query and one or more current events.